Head mounted display device, method of controlling head mounted display device, system, synchronous control apparatus, and method of controlling synchronous control apparatus

ABSTRACT

A head mounted display device including a first image capturing unit and a second image capturing unit different from the first image capturing unit comprises a first generation unit configured to generate, based on a signal representing an image output timing of the first image capturing unit, a first signal for controlling a start of exposure of the second image capturing unit, and supply the generated first signal to the second image capturing unit, and a second generation unit configured to generate, based on the signal representing the image output timing of the first image capturing unit, a second signal for controlling a start of measurement of a sensor that measures a position and orientation of the sensor, and supply the generated second signal to the sensor.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a mixed reality presentation technique.

Description of the Related Art

These days, an MR (mixed Reality) technique is known as a technique ofseamlessly blending physical and virtual worlds in real time. One of MRtechniques is an MR system using a video see-through HMD (Head MountedDisplay: to be referred to as an “HMD” if necessary). In the MR system,an HMD-incorporated image capturing unit captures an image of an objectalmost coincident with an object observed from the pupil position of anHMD wearer. Then, CG (Computer Graphics) is superimposed and displayedon the captured image, and the resultant image is presented to the HMDwearer. The HMD wearer can experience the MR space.

The MR system obtains the position and orientation of the HMD byperforming arithmetic processing using captured images and various kindsof sensor information. Image capturing units and various sensorsdesirably operate as synchronously as possible. For example, JapanesePatent Laid-Open No. 2000-341719 discloses a technique of establishingsynchronization by supplying a common driving signal and sync signal toa plurality of image capturing units. Japanese Patent Laid-Open No.2006-005608 discloses a technique of establishing synchronization bymaking the barycenters of exposure times coincide with each other for aplurality of image capturing units having different exposure times.

However, the conventional techniques have the following problems. In anarrangement typified by Japanese Patent Laid-Open Nos. 2000-341719 and2006-005608, only image capturing units are synchronized. In a systemusing various sensors in addition to image capturing units, like the MRsystem, if the image capturing units and the sensors operateasynchronously, no satisfactory arithmetic accuracy may be obtained. Inthis case, misalignment may occur between a captured image and CG.

SUMMARY OF THE INVENTION

The present invention provides a technique for synchronously operatingan image capturing unit and an orientation sensor.

According to the first aspect of the present invention, there isprovided a head mounted display device including a first image capturingunit and a second image capturing unit different from the first imagecapturing unit, comprising: a first generation unit configured togenerate, based on a signal representing an image output timing of thefirst image capturing unit, a first signal for controlling a start ofexposure of the second image capturing unit, and supply the generatedfirst signal to the second image capturing unit: and a second generationunit configured to generate, based on the signal representing the imageoutput timing of the first image capturing unit, a second signal forcontrolling a start of measurement of a sensor that measures a positionand orientation of the sensor, and supply the generated second signal tothe sensor.

According to the second aspect of the present invention, there isprovided a method of controlling a head mounted display device includinga first image capturing unit and a second image capturing unit differentfrom the first image capturing unit, the method comprising: generating,based on a signal representing an image output timing of the first imagecapturing unit, a first signal for controlling a start of exposure ofthe second image capturing unit, and supplying the generated firstsignal to the second image capturing unit: and generating, based on thesignal representing the image output timing of the first image capturingunit, a second signal for controlling a start of measurement of a sensorthat measures a position and orientation of the head mounted displaydevice, and supplying the generated second signal to the sensor.

According to the third aspect of the present invention, there isprovided a system comprising a head mounted display device including afirst image capturing unit and a second image capturing unit differentfrom the first image capturing unit, and an image processing apparatus,the head mounted display device including: a first generation unitconfigured to generate, based on a signal representing an image outputtiming of the first image capturing unit, a first signal for controllinga start of exposure of the second image capturing unit, and supply thegenerated first signal to the second image capturing unit; and a secondgeneration unit configured to generate, based on the signal representingthe image output timing of the first image capturing unit, a secondsignal for controlling a start of measurement of a sensor that measuresa position and orientation of the sensor, and supply the generatedsecond signal to the sensor, and the image processing apparatusincluding: an obtaining unit configured to obtain, from the head mounteddisplay device, an image captured by the first image capturing unit, animage captured by the second image capturing unit in response toreception of the first signal, and a position and orientation measuredby the sensor in response to reception of the second signal; a unitconfigured to generate an image of a virtual space based on the imagecaptured by the second image capturing unit and the position andorientation measured by the sensor, and generate a composite image ofthe generated image of the virtual space and the image captured by thefirst image capturing unit; and a unit configured to output thecomposite image to the head mounted display device.

According to the fourth aspect of the present invention, there isprovided a synchronous control apparatus comprising: an image capturingunit: a sensor configured to measure a position and orientation of thesensor; an obtaining unit configured to obtain a signal out of anexternal sync input signal externally input to control the imagecapturing unit, and a sync output signal output from the image capturingunit: and a control unit configured to control to execute measurementprocessing of the sensor at an arbitrary timing in an exposure time inthe image capturing unit based on the signal obtained by the obtainingunit.

According to the fifth aspect of the present invention, there isprovided a method of controlling a synchronous control apparatusincluding: an image capturing unit: and a sensor configured to measure aposition and orientation of the sensor, the method comprising: obtaininga signal out of an external sync input signal externally input tocontrol the image capturing unit, and a sync output signal output fromthe image capturing unit; and controlling to execute measurementprocessing of the sensor at an arbitrary timing in an exposure time inthe image capturing unit based on the obtained signal.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view exemplifying the arrangement of an MR system;

FIG. 2 is a view for explaining processing of generating a compositeimage from a captured image and an image of a virtual space;

FIG. 3 is a timing chart for explaining the operation timings of animage capturing unit and orientation sensor:

FIG. 4 is a timing chart for explaining the synchronization between theimage capturing unit and the orientation sensor by an external syncinput;

FIG. 5 is a block diagram exemplifying the functional arrangements of anHMD 101 and an image processing apparatus 104:

FIG. 6 is a timing chart showing an example of generating by ageneration unit 506 a sync signal to be supplied to an image capturingunit 502, and generation of an external sync input to an image capturingunit 2 when an external sync input to an image capturing unit 1 is setas a reference;

FIG. 7 is a timing chart for explaining the synchronous operation of animage capturing unit 501, the image capturing unit 502, and anorientation sensor 503:

FIG. 8 is a flowchart of control of the synchronous operation of theimage capturing unit 501, the image capturing unit 502, and theorientation sensor 503 by the HMD 101;

FIG. 9 is a block diagram exemplifying the functional arrangements ofthe HMD 101 and the image processing apparatus 104;

FIG. 10 is a timing chart for explaining generation processing ofgenerating sync signals to be supplied to the image capturing unit 502and the orientation sensor 503;

FIG. 11 is a timing chart for explaining generation processing ofgenerating sync signals to be supplied to the image capturing unit 2 andthe orientation sensor;

FIG. 12 is a timing chart for explaining generation processing ofgenerating sync signals to be supplied to the image capturing unit 2 andthe orientation sensor:

FIG. 13 is a timing chart for explaining generation processing ofgenerating sync signals to be supplied to the image capturing unit 2 andthe orientation sensor;

FIG. 14 is a flowchart of control of the synchronous operation of theimage capturing unit 501, the image capturing unit 502, and theorientation sensor 503 by the HMD 101;

FIG. 15A is a block diagram exemplifying the hardware arrangement of theHMD 101: and

FIG. 15B is a block diagram exemplifying the hardware arrangement of acomputer apparatus.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention. Multiple features aredescribed in the embodiments, but limitation is not made to an inventionthat requires all such features, and multiple such features may becombined as appropriate.

Furthermore, in the attached drawings, the same reference numerals aregiven to the same or similar configurations, and redundant descriptionthereof is omitted.

First Embodiment

First, the arrangement of an MR system according to the first embodimentwill be exemplified with reference to FIG. 1 . As shown in FIG. 1 , theMR system according to the embodiment includes an HMD 101 serving as anexample of a head mounted display device, a computer apparatus 103 thatgenerates an image of a mixed reality space (space obtained by blendingphysical and virtual spaces) to be displayed on the HMD 101, and acontroller 102 that mediates between the HMD 101 and the computerapparatus 103.

First, the HMD 101 will be described. The HMD 101 includes an imagecapturing unit that captures an image of a physical space, a sensor thatmeasures (measurement processing) the position and orientation of theHMD 101, and a display unit that displays an image of a mixed realityspace transmitted from an image processing apparatus 104. The HMD 101also functions as a synchronous control apparatus for these devices. TheHMD 101 transmits to the controller 102 an image captured by the imagecapturing unit and the position and orientation of the HMD 101 measuredby the sensor. The HMD 101 receives from the controller 102 an image ofthe mixed reality space generated by the computer apparatus 103 based onthe captured image and the position and orientation, and displays thereceived image on the display unit. The image of the mixed reality spaceis presented in front of the eyes of a user wearing the HMD 101 onhis/her head.

The HMD 101 may operate by power supplied from the controller 102 or bypower supplied from the battery of the HMD 101. That is, a method ofsupplying power to the HMD 101 is not limited to a specific one.

In FIG. 1 , the HMD 101 and the controller 102 are connected by wire.However, the connection form between the HMD 101 and the controller 102is not limited to wire, and may be wireless or a combination of wirelessand wire. That is, the connection form between the HMD 101 and thecontroller 102 is not limited to a specific one.

Next, the controller 102 will be described. The controller 102 performsvarious image processes (for example, resolution conversion, color spaceconversion, distortion correction of the optical system of the imagecapturing unit of the HMD 101, and encoding) on a captured imagetransmitted from the HMD 101. Then, the controller 102 transmits to thecomputer apparatus 103 the captured image having undergone the imageprocesses and a position and orientation transmitted from the HMD 101.The controller 102 performs similar image processes on an image of themixed reality space transmitted from the computer apparatus 103, andtransmits the processed image to the HMD 101.

Next, the computer apparatus 103 will be described. The computerapparatus 103 obtains the position and orientation (position andorientation of the image capturing unit of the HMD 101) of the HMD 101based on a captured image and a position and orientation received fromthe controller 102, and generates an image of a virtual space viewedfrom a viewpoint having the obtained position and orientation. Thecomputer apparatus 103 generates a composite image (image of the mixedreality space) of the image of the virtual space and the captured imagetransmitted from the HMD 101 via the controller 102, and transmits thegenerated composite image to the controller 102.

Processing of generating a composite image from a captured image and animage of the virtual space will be explained with reference to FIG. 2 .A captured image 201 includes a marker 202 (the number of markers is onein FIG. 2 for descriptive convenience, but a plurality of markers areincluded in practice) artificially arranged in the physical space. Thecomputer apparatus 103 extracts the marker 202 from the captured image201, and obtains the position and orientation of the HMD 101 based onthe extracted marker 202 and a position and orientation received fromthe controller 102. The computer apparatus 103 then generates an image203 of the virtual space viewed from a viewpoint having the obtainedposition and orientation. The image 203 includes a virtual object 204.

The computer apparatus 103 generates an image 205 of the mixed realityspace as a composite image of the captured image 201 and the image 203of the virtual space. The computer apparatus 103 transmits the generatedimage 205 to the HMD 101. Note that the captured image and the image ofthe virtual space are composited using information about the depth inthe 3D space and information about the transparency of a virtual object.This enables generating a composite image considering the positionalrelationship in depth between a physical object and the virtual object,or a composite image in which the virtual object is composited in asemitransparent state.

The computer apparatus 103 and the controller 102 are separateapparatuses in FIG. 1 , but may be integrated. In the embodiment, a formwill be explained in which the computer apparatus 103 and the controller102 are integrated. An apparatus constituted by integrating the computerapparatus 103 and the controller 102 will be referred to as the imageprocessing apparatus 104.

The image capturing unit of the HMD can selectively use a rollingshutter image sensor and a global shutter image sensor in considerationof various factors such as the number of pixels, image quality, noise,sensor size, power consumption, and cost, or can use them in combinationdepending on the intended use. For example, the rolling shutter imagesensor capable of obtaining a higher-quality image is used to capture animage that is composited with an image of the virtual space, and theglobal shutter image sensor free from image flow is used to capture animage of a marker. The image flow is a phenomenon arising from a rollingshutter operation principle that starts exposure processing sequentiallyfor respective lines in the scanning direction. More specifically, theimage flow is known as a phenomenon in which a time lag is generatedbetween the timings of exposure of respective lines, as shown in FIG. 3, and when an image capturing unit or an object moves during theexposure time, the object is deformed and recorded as if it flowed. InFIG. 3 , the abscissa represents the time, and “exposure time (rolling)”represents the exposure time of each line subjected to image capturingby the rolling shutter image sensor. “Exposure time (global)” representsthe exposure time of each line subjected to image capturing by theglobal shutter image sensor. In the global shutter type, exposureprocessing is performed simultaneously on all lines, so no time lag isgenerated between the exposure timings of respective lines and no imageflow occurs.

As shown in FIG. 3 , when the image capturing unit receives an externalsync input (external sync input signal), exposure of each pixel startsafter a processing time t_(Exp_Ready) till the start of exposure, andexposure of each pixel is performed during an exposure time t_(Exp) setin the image capturing unit. Output of a sync signal (sync output, syncoutput signal) and an image signal (image output) from the imagecapturing unit starts t_(Img_Out) after the end of exposure of eachline. As shown in FIG. 3 , in the global shutter image capturing unit,the time till an exposure start time t_(Exp_Start) after a timet_(Sync_In) at which an external sync input is input, an exposure centertime t_(Exp_Center), and an exposure end time t_(Exp_End) are uniquelydetermined as values unique to each image sensor. To the contrary, inthe rolling shutter image capturing unit, the exposure start time, theexposure center time, and the exposure end time can take differentvalues within a range equivalent to an exposure start timing differenceΔt_(Exp_Start) between start and final lines depending on a line whoseexposure time is set as a reference.

As for a sensor (orientation sensor) that measures the position andorientation of the HMD, when an external sync input is received,measurement (data obtainment) of the position and orientation startsafter a processing time t_(Sens_Ready) till the start of measurement,and output (data output) of the position and orientation starts a timet_(Sense_Out) after the end of measurement.

FIG. 4 is a timing chart for explaining synchronization between theimage capturing unit and the orientation sensor based on an externalsync input. In FIG. 4 , devices are synchronized by supplying a commonexternal sync input at the time t_(Sync_In) to a rolling shutter imagecapturing unit 1, a global shutter image capturing unit 2, and theorientation sensor. As described with reference to FIG. 3 , the imagecapturing unit 1, the image capturing unit 2, and the orientation sensorhave unique processing times, so lags are generated in the time when theimage capturing unit actually performs exposure, and the time when theorientation sensor performs measurement in response to an external syncinput. For example, when the exposure center time of the image capturingunit 1 or 2 is regarded as a reference, the position and orientationmeasurement time of the orientation sensor is t_(Sens_Meas), theexposure center time of the image capturing unit 1 ist_(Exp_Center_CAM1), and the exposure center time of the image capturingunit 2 is t_(Exp_Center_CAM2). These times do not coincide with eachother.

An arrangement will be considered in which the image capturing unit 1 isused to obtain a captured image that is composited with an image of thevirtual space, and the image capturing unit 2 is used to capture animage of a marker. An image of the virtual space is generated based on acaptured image of the marker obtained by the image capturing unit 2 anda position and orientation obtained by the orientation sensor, and isinfluenced by an error arising from the lag between the obtainingtimings of the captured image and the position and orientation. Further,the obtaining timing of a captured image obtained by the image capturingunit 1 differs from the obtaining timing of the captured image by theimage capturing unit 2 and the obtaining timing of the position andorientation by the orientation sensor. The influence on the positionalaccuracy of the image of the virtual space superimposed on the capturedimage becomes larger. To prevent this, it can be said that it isdesirable that the MR system performs a synchronous operation on theimage capturing unit 1, the image capturing unit 2, and the orientationsensor so that the exposure timing of the image capturing unit 1, theexposure timing of the image capturing unit 2, and the position andorientation obtaining timing of the orientation sensor coincide witheach other at higher precision.

Next, the functional arrangements of the HMD 101 and the imageprocessing apparatus 104 will be exemplified with reference to the blockdiagram of FIG. 5 . First, the HMD 101 will be explained. An imagecapturing unit 501 captures an image of the physical space that iscomposited with an image of the virtual space. The image capturing unit501 includes a left-eye image capturing portion and a right-eye imagecapturing portion. The left-eye image capturing portion captures amoving image of the physical space corresponding to the left eye of thewearer of the HMD 101, and outputs an image (captured image) of eachframe in the moving image. The right-eye image capturing portioncaptures a moving image of the physical space corresponding to the righteye of the wearer of the HMD 101, and outputs an image (captured image)of each frame in the moving image. That is, the image capturing unit 501obtains captured images as stereo images having a parallax that almostcoincide with the positions of the left and right eyes of the wearer ofthe HMD 101. Note that in the HMD of the MR system, the central opticalaxis of the image capturing range of the image capturing unit isdesirably arranged to almost coincide with the line-of-sight directionof the wearer of the HMD.

Each of the left- and right-eye image capturing portions includes anoptical system and an image capturing device. Light incoming from theoutside enters the image capturing device via the optical system, andthe image capturing device outputs an image corresponding to theentering light as a captured image. As the image capturing device of theimage capturing unit 501, a rolling shutter image sensor is used. Theimage capturing unit 501 periodically outputs a captured image and alsooutputs a sync signal representing the output start timing (image outputtiming) of the captured image.

An image capturing unit 502 includes a plurality of image capturingportions for capturing a marker arranged in the physical space, andobtains captured images as stereo images having a parallax. Each imagecapturing portion captures a moving image of the physical space andoutputs an image (captured image) of each frame in the moving image.Each image capturing portion of the image capturing unit 502 includes anoptical system and an image capturing device. Light incoming from theoutside enters the image capturing device via the optical system, andthe image capturing device outputs an image corresponding to theentering light as a captured image. As the image capturing device of theimage capturing unit 502, a global shutter image sensor is used. Aplurality of image capturing portions of the image capturing unit 502start exposure every time they receive a sync signal from a generationunit 506, and end the exposure after the lapse of an exposure time ofone frame.

An orientation sensor 503 measures the position and orientation of theHMD 101 every time it receives a sync signal from the generation unit506, and outputs the measured position and orientation. The orientationsensor 503 is implemented by a magnetic sensor, an ultrasonic sensor, anacceleration sensor, an angular velocity sensor, or the like.

A display unit 504 includes a right-eye display portion and a left-eyedisplay portion. The left-eye display portion displays a left-eye imageof the mixed reality space received from the image processing apparatus104 via an/F 508. The right-eye display portion displays a right-eyeimage of the mixed reality space received from the image processingapparatus 104 via the I/F 508. Each of the left- and right-eye displayportions includes a display optical system and a display element. Thedisplay optical system may be not only an eccentric optical system suchas a free-form surface prism, but also a normal co-axial optical systemor an optical system having a zoom mechanism. The display element is,for example, a compact liquid crystal display, an organic EL display, ora MEMS retina scanning device. Light traveling from an image displayedon the display element enters the eyes of the wearer of the HMD 101 viathe display optical system.

A detection unit 505 detects a sync signal (signal representing thestart timing of image output from the image capturing unit 501) outputfrom the image capturing unit 501, and upon detecting the sync signal,notifies the generation unit 506 of the detection.

When the generation unit 506 receives the notification from thedetection unit 505, it generates sync signals to be supplied to theimage capturing unit 502 and the orientation sensor 503 based on thesync signal detected by the detection unit 505, and supplies thegenerated sync signals to the image capturing unit 502 and theorientation sensor 503. A setting unit 507 sets various parameters usedin the operation of the HMD 101.

All of a captured image output from the image capturing unit 501, acaptured image output from the image capturing unit 502, and a positionand orientation output from the orientation sensor 503 are transmittedto the image processing apparatus 104 via the I/F 508.

Next, the image processing apparatus 104 will be described. The imageprocessing apparatus 104 receives via an I/F 509 the captured images andthe position and orientation transmitted from the HMD 101. A processingunit 510 performs various image processes on the captured imagesreceived from the HMD 101 via the I/F 509.

A generation unit 511 extracts (recognizes) markers from the left- andright-eye captured images having undergone the image processes by theprocessing unit 510. The generation unit 511 obtains the positions andorientations of the left- and right-eye image capturing portions basedon the markers and the position and orientation received from the HMD101 via the I/F 509. Processing for obtaining the position andorientation of an image capturing portion based on a marker in an image,and a position and orientation measured by a sensor included in an HMDtogether with the image capturing unit is generally known, so adescription of this technique will be omitted.

Various data (virtual space data) necessary to render an image of thevirtual space is saved in a content DB (DataBase) 512. The virtual spacedata includes, for example, data defining each virtual objectconstituting the virtual space (for example, data defining the geometricshape, color, texture, arrangement position and orientation, and thelike of the virtual object). Also, the virtual space data includes, forexample, data defining a light source arranged in the virtual space (forexample, data defining the type, position and orientation, and the likeof the light source).

A composition unit 513 builds a virtual space using the virtual spacedata saved in the content DB 512. The composition unit 513 generates animage L of the virtual space viewed from a viewpoint having the positionand orientation of the left-eye image capturing portion obtained by thegeneration unit 511. The composition unit 513 generates an image R ofthe virtual space viewed from a viewpoint having the position andorientation of the right-eye image capturing portion obtained by thegeneration unit 511. The composition unit 513 generates a compositeimage L as a left-eye image L of the mixed reality space by compositingthe image L of the virtual space and an image captured by the left-eyeimage capturing portion. The composition unit 513 generates a compositeimage R as a right-eye image R of the mixed reality space by compositingthe image R of the virtual space and an image captured by the right-eyeimage capturing portion.

A processing unit 514 performs various image processes on the image L ofthe mixed reality space and the image R of the mixed reality spacegenerated by the composition unit 513. The processing unit 514transmits, to the HMD 101 via the I/F 509, the image L of the mixedreality space and the image R of the mixed reality space havingundergone the image processes. A setting unit 515 sets variousparameters used in the operation of the image processing apparatus 104.

Next, an example of generating by the generation unit 506 a sync signalto be supplied to the image capturing unit 502 will be explained withreference to a timing chart on the upper part of FIG. 6 . In thefollowing description, an “image capturing unit 1” corresponds to theimage capturing unit 501, and an “image capturing unit 2” corresponds tothe image capturing unit 502. In FIG. 6 , the abscissa represents thetime.

When the detection unit 505 detects a sync signal (sync output) from theimage capturing unit 1, it notifies the generation unit 506 of thedetection. Upon receiving the notification, the generation unit 506generates a “sync signal (external sync input) for controlling the startof exposure of the image capturing unit 2” after an offset 601 from thedetection timing of the sync signal. Here, t61 is the time till a“center time (exposure center time) in the exposure time of the nextframe at the detection timing in the image capturing unit 1” after thedetection timing of the sync signal, t62 is a “processing time till thestart of exposure” unique to the image capturing unit 2, and t63 is theexposure time of one frame in the image capturing unit 2. At this time,the offset 601 can be calculated according to the following equation:offset 601=t61−t62−t63/2

Assume that the offset 601 is obtained and set in advance by the settingunit 507. Note that the setting unit 507 may obtain and set the offset601 periodically or irregularly. The generation unit 506 supplies thegenerated “sync signal for controlling the start of exposure of theimage capturing unit 2” to the image capturing unit 2.

Upon receiving the generated “sync signal for controlling the start ofexposure of the image capturing unit 2”, the image capturing unit 2starts exposure. Since the center time of the exposure time coincideswith the center time in the exposure time of the image capturing unit 1,the image capturing units 1 and 2 perform exposure synchronously as aresult. That is, the generation unit 506 generates a sync signal to besupplied to the image capturing unit 2 so that the center time of theexposure time of the image capturing unit 1 coincides with that of theexposure time of the image capturing unit 2.

Even if devices to perform a synchronous operation include a devicehaving no external sync input function, the device is set as thereference of the synchronous operation and the synchronous operationbetween the devices can be implemented.

A timing chart on the lower part of FIG. 6 shows generation of anexternal sync input to the image capturing unit 2 when an external syncinput to the image capturing unit 1 is set as a reference. When the HMD101 detects an external sync input to the image capturing unit 1, itgenerates an external sync input to the image capturing unit 2 after anoffset 601′. Letting 64 be the time till a “center time in the exposuretime of the next frame at the detection timing in the image capturingunit 1” after the detection timing of the external sync input to theimage capturing unit 1, the offset 601′ can be calculated according tothe following equation:offset 601′=t64−t62−t63/2

In the above description, the synchronous timing in the synchronousoperation is the center time in the exposure time of the image capturingunit 1.

However, the setting of the synchronous timing in the embodiment is notlimited to this and can be an arbitrary timing in the exposure time. Asynchronous operation when a sync signal from the image capturing unit 1in the timing chart shown in the upper part of FIG. 6 is set as areference will be described. However, the synchronous operation can beimplemented similarly even when an external sync input to the imagecapturing unit 1 in the timing chart shown in the lower part of FIG. 6is set as a reference.

The synchronous operation of the image capturing unit 501, the imagecapturing unit 502, and the orientation sensor 503 according to theembodiment will be described with reference to FIG. 7 . In FIG. 7 , theabscissa represents the time.

When the detection unit 505 detects a sync signal (sync output) from theimage capturing unit 1, it notifies the generation unit 506 of thedetection. Upon receiving the notification, the generation unit 506generates a “sync signal (external sync input) for controlling the startof exposure of the image capturing unit 2” after an offset 701 from thedetection timing of the sync signal. The offset 701 is set by thesetting unit 507 similarly to the offset 601. Upon receiving thegenerated “sync signal for controlling the start of exposure of theimage capturing unit 2”, the image capturing unit 2 starts exposure andoutputs, in accordance with the sync output of the image capturing unit2, data of each line captured by the exposure.

Upon receiving the notification, the generation unit 506 generates a“sync signal (external sync input) for controlling the start ofmeasurement of a position and orientation by the orientation sensor 503”after an offset 702 from the detection timing of the sync signal. Theoffset 702 can be calculated using the time t61 and a “processing timet_(Sens_Ready) till the start of measurement of a position andorientation after the orientation sensor 503 receives an external syncinput” according to the following equation:offset 702=t61−t _(Sens_Ready)

Assume that the offset 702 is obtained and set in advance by the settingunit 507. The generation unit 506 supplies the generated “sync signalfor controlling the start of measurement of a position and orientationby the orientation sensor 503” to the orientation sensor 503. Uponreceiving the generated “sync signal for controlling the start ofmeasurement of a position and orientation by the orientation sensor503”, the orientation sensor 503 starts measurement (data obtainment) ofa position and orientation. That is, the generation unit 506 generates async signal to be supplied to the image capturing unit 2 and a syncsignal to be supplied to the orientation sensor 503 so that the centertime of the exposure time of the image capturing unit 1, that of theexposure time of the image capturing unit 2, and the timing (measurementtiming) of data obtainment by the orientation sensor 503 coincide witheach other.

Control of the synchronous operation of the image capturing unit 501,the image capturing unit 502, and the orientation sensor 503 by the HMD101 according to the embodiment will be described with reference to theflowchart of FIG. 8 . If the detection unit 505 detects a sync signal(sync output) from the image capturing unit 501, it notifies thegeneration unit 506 of the detection and the process advances to stepS802 via step S801. If the detection unit 505 does not detect a syncsignal (sync output) from the image capturing unit 501, the processstands by in step S801.

In step S802, upon receiving the notification from the detection unit505, the generation unit 506 generates a “sync signal to be supplied tothe image capturing unit 502” after the offset 701 from the detectiontiming of the sync signal, and supplies the generated sync signal to theimage capturing unit 502. Further, the generation unit 506 generates a“sync signal to be supplied to the orientation sensor 503” after theoffset 702 from the detection timing of the sync signal, and suppliesthe generated sync signal to the orientation sensor 503.

As described above, according to the first embodiment, the synchronousoperation of the image capturing unit 501, the image capturing unit 502,and the orientation sensor 503 can be implemented based on the detectiontiming of a sync signal from the image capturing unit 501 or an externalsync input to the image capturing unit 501. Since the processing timesand the like of respective devices are considered, the image capturingand data obtaining timings of the respective devices can coincide withan arbitrary timing in the exposure time. A more real MR experience freefrom misalignment between a captured image and an image of the virtualspace can be provided. Even when the image capturing unit 501 does nothave the external sync input function, the synchronous operation can beimplemented and the choice of devices can be widened.

Second Embodiment

In the following embodiments including the second embodiment,differences from the first embodiment will be explained, and theremaining parts are similar to the first embodiment, unless otherwisespecified. In the first embodiment, an arrangement has been described inwhich sync signals to respective devices are generated in considerationof the processing times and the like of the respective devices withreference to the detection timing of a sync signal from the imagecapturing unit 501. In the second embodiment, an arrangement will beexplained in which the synchronous operation between devices isperformed when the setting of the exposure time of an image capturingunit 501 or 502 is changed or the setting of the synchronous timing ischanged.

First, the functional arrangements of an HMD 101 and an image processingapparatus 104 will be exemplified with reference to the block diagram ofFIG. 9 . A generation unit 506 generates a sync signal to be supplied tothe image capturing unit 502 and a sync signal to be supplied to anorientation sensor 503 under the control of a synchronous control unit901.

The synchronous control unit 901 controls generation of a sync signal bythe generation unit 506 in accordance with a change of the setting ofthe exposure time of the image capturing unit 501 or 502 by a settingunit 507. The synchronous control unit 901 controls generation of a syncsignal by the generation unit 506 in accordance with a change of thesetting of a time (line corresponding to the time) in the exposure timeof the image capturing unit 501 with which the generation timing of async signal to be supplied to the image capturing unit 502 or theorientation sensor 503 is synchronized.

Next, generation processing of generating sync signals to be supplied tothe image capturing unit 502 and the orientation sensor 503 inconsideration of a lag between the exposure times of lines of an imagecaptured by a rolling shutter image sensor will be described withreference to FIG. 10 .

When a detection unit 505 detects a sync signal (sync output) from animage capturing unit 1, it notifies the generation unit 506 of thedetection. Upon receiving the notification, the generation unit 506generates a sync signal to be supplied to an image capturing unit 2 anda sync signal to be supplied to the orientation sensor 503 in accordancewith the detection timing of the sync signal under the control of thesynchronous control unit 901.

Assume that the exposure time of the image capturing unit 1 and that ofthe image capturing unit 2 are equal, and the setting unit 507 sets the“exposure start time of the start line of an image captured by the imagecapturing unit 1 as the reference of the synchronous timing”. In thiscase, the generation unit 506 generates a “sync signal (external syncinput) for controlling the start of exposure of the image capturing unit2” after an offset 1001 from the detection timing of the sync signal.The offset 1001 can be obtained as a result of subtracting the time t62from a time till the “exposure start time of the start line of an imagecaptured by the image capturing unit 1” after the detection timing ofthe sync signal. Also, the generation unit 506 generates a “sync signal(external sync input) for controlling the start of measurement of aposition and orientation by the orientation sensor” after an offset 1004from the detection timing of the sync signal. The offset 1004 can beobtained as a result of subtracting the processing time tSens_Ready froma time till the “exposure start time of the start line of an imagecaptured by the image capturing unit 1” after the detection timing ofthe sync signal.

Assume that the exposure time of the image capturing unit 1 and that ofthe image capturing unit 2 are equal, and the setting unit 507 sets the“exposure center time of the center line of an image captured by theimage capturing unit 1 as the reference of the synchronous timing”. Inthis case, the generation unit 506 generates a “sync signal (externalsync input) for controlling the start of exposure of the image capturingunit 2” after an offset 1002 from the detection timing of the syncsignal. The offset 1002 can be obtained as a result of subtracting thetime t62 from a time till the “exposure start time of the center line ofan image captured by the image capturing unit 1” after the detectiontiming of the sync signal. Also, the generation unit 506 generates a“sync signal (external sync input) for controlling the start ofmeasurement of a position and orientation by the orientation sensor”after an offset 1005 from the detection timing of the sync signal. Theoffset 1005 can be obtained as a result of subtracting the processingtime tSens_Ready from a time till the “exposure start time of the centerline of an image captured by the image capturing unit 1” after thedetection timing of the sync signal.

Assume that the exposure time of the image capturing unit 1 and that ofthe image capturing unit 2 are equal, and the setting unit 507 sets the“exposure end time of the final line of an image captured by the imagecapturing unit 1 as the reference of the synchronous timing”. In thiscase, the generation unit 506 generates a “sync signal (external syncinput) for controlling the start of exposure of the image capturing unit2” after an offset 1003 from the detection timing of the sync signal.The offset 1003 can be obtained as a result of subtracting the time t62from a time till the “exposure start time of the final line of an imagecaptured by the image capturing unit 1” after the detection timing ofthe sync signal. Also, the generation unit 506 generates a “sync signal(external sync input) for controlling the start of measurement of aposition and orientation by the orientation sensor” an offset 1006 afterthe detection timing of the sync signal. The offset 1006 can be obtainedas a result of subtracting the processing time tSens_Ready from a timetill the “exposure start time of the final line of an image captured bythe image capturing unit 1” after the detection timing of the syncsignal.

In this manner, the synchronous control unit 901 supplies offsetscorresponding to the reference of the synchronous timing to thegeneration unit 506 so that sync signals to be supplied to the imagecapturing unit 2 and the orientation sensor are generated based on theoffsets corresponding to the reference of the synchronous timing.

As for arbitrary timings such as the exposure start time of the finalline and the exposure end time of the start line, the synchronousoperation can be performed by adjusting offsets to the image capturingunit 2 and the orientation sensor based on various setting contents bythe setting unit 507.

Next, generation processing of generating sync signals to be supplied tothe image capturing unit 2 and the orientation sensor using, as thereference of the synchronous timing, the exposure center time of thecenter line of an image captured by a rolling shutter image sensor willbe described with reference to FIG. 11 .

When the detection unit 505 detects that the image capturing unit 1 hasoutput a sync signal corresponding to a captured image (frame (N−1)) ofthe (N−1)th frame, the generation unit 506 generates a “sync signal(external sync input) for controlling the start of exposure (for imagecapturing of a frame N) of the image capturing unit 2” after an offset1101, and supplies it to the image capturing unit 2. The offset 1101 canbe obtained by the calculation method described with reference to FIG.10 (offset calculation method when the “exposure center time of thecenter line of an image captured by the image capturing unit 1 is set asthe reference of the synchronous timing”). When the detection unit 505detects that the image capturing unit 1 has output a sync signalcorresponding to the frame (N−1), the generation unit 506 generates a“sync signal (external sync input) for controlling the start ofmeasurement (for the frame N) of a position and orientation by theorientation sensor” after an offset 1104, and supplies it to theorientation sensor. The offset 1104 can be obtained by the calculationmethod described with reference to FIG. 10 (offset calculation methodwhen the “exposure center time of the center line of an image capturedby the image capturing unit 1 is set as the reference of the synchronoustiming”).

Assume that the setting unit 507 performs a setting (setting change 2)of changing the exposure time of the image capturing unit 2 until thedetection unit 505 detects a sync signal corresponding to the frame Nafter detecting a sync signal corresponding to the frame (N−1).

At this time, when the detection unit 505 detects that the imagecapturing unit 1 has output a sync signal corresponding to the frame N,the generation unit 506 generates a “sync signal (external sync input)for controlling the start of exposure (for image capturing of a frame(N+1)) of the image capturing unit 2” after an offset 1102, and suppliesit to the image capturing unit 2. The offset 1102 can be obtained by amethod similar to that of the offset 1101. Further, when the detectionunit 505 detects that the image capturing unit 1 has output a syncsignal corresponding to the frame N, the generation unit 506 generates a“sync signal (external sync input) for controlling the start ofmeasurement (for the frame (N+1)) of a position and orientation by theorientation sensor” after an offset 1104, and supplies it to theorientation sensor.

Assume that the setting unit 507 performs a setting (setting change 1)of changing the exposure time of the image capturing unit 1 until thedetection unit 505 detects a sync signal corresponding to the frame(N+1) after detecting a sync signal corresponding to the frame N.

At this time, when the detection unit 505 detects that the imagecapturing unit 1 has output a sync signal corresponding to the frame(N+1), the generation unit 506 generates a “sync signal (external syncinput) for controlling the start of exposure (for image capturing of aframe (N+2)) of the image capturing unit 2” after an offset 1103, andsupplies it to the image capturing unit 2. The offset 1103 can beobtained by a method similar to that of the offset 1101. Further, whenthe detection unit 505 detects that the image capturing unit 1 hasoutput a sync signal corresponding to the frame (N+1), the generationunit 506 generates a “sync signal (external sync input) for controllingthe start of measurement (for the frame (N+2)) of a position andorientation by the orientation sensor” after an offset 1105, andsupplies it to the orientation sensor. The offset 1105 can be obtainedby a method similar to that of the offset 1104.

After that, no setting change by the setting unit 507 is performed, andno offset switching occurs. In this way, the synchronous control unit901 obtains offsets and supplies them to the generation unit 506 so thatsync signals to be supplied to the image capturing unit 2 and theorientation sensor are generated based on the offsets corresponding to asetting change of the exposure time of the image capturing unit by thesetting unit 507.

The above-described processing can be performed to implement asynchronous operation in which the exposure center time of the imagecapturing unit 1, that of the image capturing unit 2, and the dataobtaining timing of the orientation sensor coincide with each other evenwhen a setting change is performed.

Next, generation processing of generating sync signals to be supplied tothe image capturing unit 2 and the orientation sensor using, as thereference of the synchronous timing, the exposure start time of thestart line of an image captured by a rolling shutter image sensor willbe described with reference to FIG. 12 .

When the detection unit 505 detects that the image capturing unit 1 hasoutput a sync signal corresponding to the frame (N−1), the generationunit 506 generates a “sync signal (external sync input) for controllingthe start of exposure (for image capturing of the frame N) of the imagecapturing unit 2” after an offset 1201, and supplies it to the imagecapturing unit 2. The offset 1201 can be obtained by the calculationmethod described with reference to FIG. 10 (offset calculation methodwhen the “exposure start time of the start line of an image captured bythe image capturing unit 1 is set as the reference of the synchronoustiming”). When the detection unit 505 detects that the image capturingunit 1 has output a sync signal corresponding to the frame (N−1), thegeneration unit 506 generates a “sync signal (external sync input) forcontrolling the start of measurement (for the frame N) of a position andorientation by the orientation sensor” after an offset 1203, andsupplies it to the orientation sensor. The offset 1203 can be obtainedby the calculation method described with reference to FIG. 10 (offsetcalculation method when the “exposure start time of the start line of animage captured by the image capturing unit 1 is set as the reference ofthe synchronous timing”).

Assume that the setting unit 507 performs the setting (setting change 2)of changing the exposure time of the image capturing unit 2 until thedetection unit 505 detects a sync signal corresponding to the frame Nafter detecting a sync signal corresponding to the frame (N−1).

Even if the setting change 2 is performed, the relationship between theexposure start time of the image capturing unit 1 and the processingtime till the exposure start time after the external sync input of theimage capturing unit 2 does not change, so the offset need not bechanged.

Hence, when the detection unit 505 detects that the image capturing unit1 has output a sync signal corresponding to the frame N, the generationunit 506 generates a “sync signal (external sync input) for controllingthe start of exposure (for image capturing of the frame (N+1)) of theimage capturing unit 2” after the offset 1201, and supplies it to theimage capturing unit 2. Further, when the detection unit 505 detectsthat the image capturing unit 1 has output a sync signal correspondingto the frame N, the generation unit 506 generates a “sync signal(external sync input) for controlling the start of measurement (for theframe (N+1)) of a position and orientation by the orientation sensor”after the offset 1203, and supplies it to the orientation sensor.

Assume that the setting unit 507 performs the setting (setting change 1)of changing the exposure time of the image capturing unit 1 until thedetection unit 505 detects a sync signal corresponding to the frame(N+1) after detecting a sync signal corresponding to the frame N.

At this time, when the detection unit 505 detects that the imagecapturing unit 1 has output a sync signal corresponding to the frame(N+1), the generation unit 506 generates a “sync signal (external syncinput) for controlling the start of exposure (for image capturing of theframe (N+2)) of the image capturing unit 2” after an offset 1202, andsupplies it to the image capturing unit 2. The offset 1202 can beobtained by a method similar to that of the offset 1201. Also, when thedetection unit 505 detects that the image capturing unit 1 has output async signal corresponding to the frame (N+1), the generation unit 506generates a “sync signal (external sync input) for controlling the startof measurement (for the frame (N+2)) of a position and orientation bythe orientation sensor” after an offset 1204, and supplies it to theorientation sensor. The offset 1204 can be obtained by a method similarto that of the offset 1203.

Thereafter, no setting change by the setting unit 507 is performed, andno offset switching occurs. In this fashion, the synchronous controlunit 901 supplies offsets to the generation unit 506 so that syncsignals to be supplied to the image capturing unit 2 and the orientationsensor are generated based on the offsets corresponding to a settingchange of the exposure time of the image capturing unit by the settingunit 507.

The above-described processing can be performed to implement asynchronous operation in which the exposure start time of the first lineof an image captured by the image capturing unit 1, that of the firstline of an image captured by the image capturing unit 2, and the dataobtaining timing of the orientation sensor coincide with each other evenwhen a setting change is performed.

In the case in which the synchronous timing is set at the exposure starttime, offsets for generating external sync inputs to the image capturingunit 2 and the orientation sensor are changed only when the settingchange 1 of the exposure time of the image capturing unit 1 isperformed. This also applies to a case in which the synchronous timingis set not only at the exposure start time of the start line but also atthe exposure start time of an arbitrary line. This can be utilized tosimplify offset change processing when a setting change is performed,and reduce the processing load and circuit scale of the HMD 101.

Next, generation processing of generating sync signals to be supplied tothe image capturing unit 2 and the orientation sensor using, as thereference of the synchronous timing, the exposure end time of the centerline of an image captured by a rolling shutter image sensor will bedescribed with reference to FIG. 13 .

When the detection unit 505 detects that the image capturing unit 1 hasoutput a sync signal corresponding to the frame (N−1), the generationunit 506 generates a “sync signal (external sync input) for controllingthe start of exposure (for image capturing of the frame N) of the imagecapturing unit 2” after an offset 1301, and supplies it to the imagecapturing unit 2. The offset 1301 can be obtained as a result ofsubtracting the “exposure time of the image capturing unit 2” and the“time t62” from a “time till the exposure end time of the center line ofan image captured by the image capturing unit 1 after the detectiontiming of the sync signal of the image capturing unit 1”. When thedetection unit 505 detects that the image capturing unit 1 has output async signal corresponding to the frame (N−1), the generation unit 506generates a “sync signal (external sync input) for controlling the startof measurement (for the frame N) of a position and orientation by theorientation sensor” after an offset 1303, and supplies it to theorientation sensor. The offset 1303 can be obtained as a result ofsubtracting the “processing time t_(Sens_Ready)” from a “time till theexposure end time of the center line of an image captured by the imagecapturing unit 1 after the detection timing of the sync signal of theimage capturing unit 1”.

Assume that the setting unit 507 performs the setting (setting change 2)of changing the exposure time of the image capturing unit 2 until thedetection unit 505 detects a sync signal corresponding to the frame Nafter detecting a sync signal corresponding to the frame (N−1).

At this time, when the detection unit 505 detects that the imagecapturing unit 1 has output a sync signal corresponding to the frame N,the generation unit 506 generates a “sync signal (external sync input)for controlling the start of exposure (for image capturing of the frame(N+1)) of the image capturing unit 2” after an offset 1302, and suppliesit to the image capturing unit 2. The offset 1302 can be obtained by amethod similar to that of the offset 1301. Further, when the detectionunit 505 detects that the image capturing unit 1 has output a syncsignal corresponding to the frame N, the generation unit 506 generates a“sync signal (external sync input) for controlling the start ofmeasurement (for the frame (N+1)) of a position and orientation by theorientation sensor” after the offset 1303, and supplies it to theorientation sensor.

Assume that the setting unit 507 performs the setting (setting change 1)of changing the exposure time of the image capturing unit 1 until thedetection unit 505 detects a sync signal corresponding to the frame(N+1) after detecting a sync signal corresponding to the frame N.

At this time, when the detection unit 505 detects that the imagecapturing unit 1 has output a sync signal corresponding to the frame(N+1), the generation unit 506 generates a “sync signal (external syncinput) for controlling the start of exposure (for image capturing of theframe (N+2)) of the image capturing unit 2” after the offset 1302, andsupplies it to the image capturing unit 2. Also, when the detection unit505 detects that the image capturing unit 1 has output a sync signalcorresponding to the frame (N+1), the generation unit 506 generates a“sync signal (external sync input) for controlling the start ofmeasurement (for the frame (N+2)) of a position and orientation by theorientation sensor” after the offset 1303, and supplies it to theorientation sensor.

After that, no setting change by the setting unit 507 is performed, andno offset switching occurs. In this manner, the synchronous control unit901 supplies offsets to the generation unit 506 so that sync signals tobe supplied to the image capturing unit 2 and the orientation sensor aregenerated based on the offsets corresponding to a setting change of theexposure time of the image capturing unit by the setting unit 507.

The above-described processing can be performed to implement asynchronous operation in which the exposure end time of the center lineof an image captured by the image capturing unit 1, that of the centerline of an image captured by the image capturing unit 2, and the dataobtaining timing of the orientation sensor coincide with each other evenwhen a setting change is performed.

In the case in which the synchronous timing is set at the exposure endtime, an offset for generating an external sync input to the imagecapturing unit 2 is changed only when the setting change 2 of theexposure time of the image capturing unit 2 is performed. In addition,in the case in which the synchronous timing is set at the exposure endtime, an offset for generating an external sync input to the orientationsensor is not changed. This also applies to a case in which thesynchronous timing is set not only at the exposure end time of thecenter line but also at the exposure end time of an arbitrary line. Thiscan be utilized to simplify offset change processing when a settingchange is performed, and reduce the processing load and circuit scale ofthe HMD 101.

Further, in the case in which the synchronous timing is set at theexposure end time, the cycle of the sync output and image output of theimage capturing unit 2 and the cycle of the data output of theorientation sensor become advantageously constant in synchronizationwith the cycle of the sync output and image output of the imagecapturing unit 1. This also applies to a case in which the synchronoustiming is set not only at the exposure end time of the center line butalso at the exposure end time of an arbitrary line.

For example, it can be confirmed in FIGS. 11 and 12 that when thesetting change 1 of the image capturing unit 1 and the setting change 2of the image capturing unit 2 are performed, the cycle of the syncoutput and image output of the image capturing unit 2 and the cycle ofthe data output of the orientation sensor vary. If a captured image isnot output in a constant cycle, an IC (Integrated Circuit) correspondingto such an output cycle needs to be selected as a subsequent capturedimage processing unit. Also, a frame memory or the like needs to beadded to adjust the image cycle constant. To the contrary, when acaptured image is output in a constant cycle, as shown in FIG. 13 , thechoice of subsequent captured image processing units is widened, and noframe memory for cycle adjustment is required.

Control of the synchronous operation of the image capturing unit 501,the image capturing unit 502, and the orientation sensor 503 by the HMD101 according to the embodiment will be described with reference to theflowchart of FIG. 14 . In step S1401, the synchronous control unit 901determines whether the setting unit 507 has performed a setting changeas described above.

If it is determined that the setting unit 507 has performed a settingchange as described above, the process advances to step S1402. If thesetting unit 507 has not performed a setting change as described above,the process advances to step S1407.

In step S1402, the synchronous control unit 901 obtains from the settingunit 507 the setting of which line of an image captured by whichcapturing unit to set as the reference of the synchronous timing. Instep S1403, the synchronous control unit 901 obtains from the settingunit 507 the setting of which time in the exposure time to set as thereference of the synchronous timing.

In step S1404, the synchronous control unit 901 obtains from the settingunit 507 the settings (including at least parameters regarding the imagecapturing unit 1 necessary to obtain an offset) of the image capturingunit 1. As described with reference to FIG. 13 , when the reference lineof the synchronous timing does not change and the reference exposuretime is the exposure end time, the obtaining processing in step S1404may be skipped because the exposure time setting of the image capturingunit 1 does not influence the decision of an offset.

In step S1405, the synchronous control unit 901 obtains from the settingunit 507 the settings (including at least parameters regarding the imagecapturing unit 2 necessary to obtain an offset) of the image capturingunit 2. As described with reference to FIG. 12 , when the reference lineof the synchronous timing does not change and the reference start timeis the exposure start time, the obtaining processing in step S1405 maybe skipped because the exposure time setting of the image capturing unit2 does not influence the decision of an offset.

In step S1406, the synchronous control unit 901 performs theabove-described processing based on the pieces of information obtainedin steps S1402 to S1406, obtaining an offset corresponding to the imagecapturing unit 2 and an offset corresponding to the orientation sensor.

In step S1407, it is determined whether the detection unit 505 hasdetected a sync signal (sync output) from the image capturing unit 501.If it is determined that the detection unit 505 has detected a syncsignal (sync output) from the image capturing unit 501, the processadvances to step S1408. If it is determined that the detection unit 505has not detected a sync signal (sync output) from the image capturingunit 501, the process returns to step S1401.

In step S1408, if the generation unit 506 receives from the detectionunit 505 a notification that the sync signal (sync output) has beenreceived from the image capturing unit 501, it generates a “sync signalto be supplied to the image capturing unit 502” after the offsetcorresponding to the image capturing unit 2 from the detection timing ofthe sync signal, and supplies the generated sync signal to the imagecapturing unit 502. Further, the generation unit 506 generates a “syncsignal to be supplied to the orientation sensor 503” after the offsetcorresponding to the orientation sensor 503 from the detection timing ofthe sync signal, and supplies the generated sync signal to theorientation sensor 503.

As described above, according to the second embodiment, the synchronousoperation of the image capturing unit 501, the image capturing unit 502,and the orientation sensor 503 can be implemented. In addition, asetting change of the exposure time of the image capturing unit 501 or502 or a setting change of the exposure timing can be coped with. Sincethe image capturing and data obtaining timings of the respective devicescoincide with an arbitrary timing in the exposure time, a more real MRexperience free from misalignment between a captured image and an imageof the virtual space can be provided.

Third Embodiment

The functional units in the HMD 101 and the image processing apparatus104 shown in FIG. 5 or 9 may be implemented by hardware, or somefunctional units may be implemented by software (computer program).

In the latter case, the image capturing unit 501, the image capturingunit 502, the orientation sensor 503, the display unit 504, and the I/F508 in the HMD 101 may be implemented by hardware, and the remainingfunctional units may be implemented by software. In this case, thesoftware is stored in the memory of the HMD 101 and executed by theprocessor of the HMD 101 to implement the functions of correspondingfunctional units.

The hardware arrangement of an HMD 101 will be exemplified withreference to the block diagram of FIG. 15A. A processor 1510 executesvarious processes using computer programs and data stored in a RAM 1520.By these processes, the processor 1510 controls the operation of thewhole HMD 101, and executes or controls each process that is performedby the HMD 101 in the above description.

The RAM 1520 has an area for storing computer programs and data loadedfrom a nonvolatile memory 1530, and an area for storing data receivedfrom an image processing apparatus 104 via an I/F 1570. Further, the RAM1520 has a work area used when the processor 1510 executes variousprocesses. The RAM 1520 can properly provide various areas.

The nonvolatile memory 1530 stores computer programs and data forcausing the processor 1510 to execute or control the operation of theHMD 101. The computer programs stored in the nonvolatile memory 1530include computer programs for causing a CPU 1501 to execute thefunctions of the functional units (except image capturing units 501 and502, an orientation sensor 503, a display unit 504, and an I/F 508) ofthe HMD 101 shown in FIG. 5 or 9 . The computer programs and data storedin the nonvolatile memory 1530 are properly loaded to the RAM 1520 underthe control of the processor 1510, and processed by the processor 1510.

An image capturing unit 1540 includes the above-described imagecapturing units 501 and 502. An orientation sensor 1550 includes theabove-described orientation sensor 503. A display unit 1560 includes theabove-described display unit 504. The I/F 1570 includes theabove-described I/F 508. All the processor 1510, the RAM 1520, thenonvolatile memory 1530, the image capturing unit 1540, the orientationsensor 1550, the display unit 1560, and the I/F 1570 are connected to abus 1580. Note that the arrangement shown in FIG. 15A is an example ofthe arrangement applicable to the HMD 101 and can be properlychanged/deformed.

As for the image processing apparatus 104, any computer apparatuscapable of executing software corresponding to functional units exceptan 1F 509 and a content DB 512 is applicable to the image processingapparatus 104. The hardware arrangement of the computer apparatusapplicable to the image processing apparatus 104 will be exemplifiedwith reference to the block diagram of FIG. 15B.

The CPU 1501 executes various processes using computer programs and datastored in a RAM 1502 and a ROM 1503. The CPU 1501 controls the operationof the whole computer apparatus, and executes or controls each processthat is performed by the computer apparatus-applied image processingapparatus 104 in the above description.

The RAM 1502 has an area for storing computer programs and data loadedfrom the ROM 1503 and an external storage device 1506, and an area forstoring data received from the HMD 101 via the I/F 1507. In addition,the RAM 1502 has a work area used when the CPU 1501 executes variousprocesses. The RAM 1502 can properly provide various areas. The ROM 1503stores setting data, startup programs, and the like of the computerapparatus.

An operation unit 1504 is a user interface including a keyboard, amouse, a touch panel, and the like. By operating the operation unit1504, the user can input various instructions to the CPU 1501.

A display unit 1505 is formed from a liquid crystal screen, a touchpanel screen, or the like, and can display the result of processing ofthe CPU 1501 using an image or a text. Note that the display unit 1505may be a projection apparatus such as a projector that projects an imageor a text.

The external storage device 1506 is amass information storage devicesuch as a hard disk drive. The external storage device 1506 stores an OS(Operating System). The external storage device 1506 stores computerprograms and data for causing the CPU 1501 to execute the functions ofthe functional units (except an I/F 509 and the content DB 512) of theimage processing apparatus 104 shown in FIG. 5 or 9 . The externalstorage device 1506 also includes the above-described content DB 512.

The computer programs and data stored in the external storage device1506 are properly loaded to the RAM 1502 under the control of the CPU1501, and processed by the CPU 1501.

An/F 1507 is a communication interface for performing data communicationwith the HMD 101 and functions as the above-described I/F 509. That is,the computer apparatus performs data communication with the HMD 101 viathe I/F 1507.

All the CPU 1501, the RAM 1502, the ROM 1503, the operation unit 1504,the display unit 1505, the external storage device 1506, and the I/F1507 are connected to a bus 1508. Note that the arrangement shown inFIG. 15B is an example of the arrangement applicable to the imageprocessing apparatus 104 and can be properly changed/deformed.

Fourth Embodiment

In each of the above-described embodiments, a marker artificiallyarranged in the physical space is used to obtain the position andorientation of an image capturing unit. However, in addition to orinstead of the marker, a natural feature (for example, the corner offurniture such as a chair or desk, or the corner of a building, car, orthe like forming a landscape) originally present in the physical spacemay be used to obtain the position and orientation of an image capturingunit.

The arrangement of the MR system shown in FIG. 5 or 9 is merely anexample. For example, a plurality of apparatuses may share and executeprocesses that are performed by the HMD 101 in the above description, orshare and execute processes that are performed by the image processingapparatus 104 in the above description.

Instead of ahead mounted display device, a “portable device includingone or more image capturing units, an orientation sensor, and a displaydevice” such as a smartphone may be used. Also, such a portable devicemay be added to the MR system together with the head mounted displaydevice. In this case, an image processing apparatus 104 generates animage of the mixed reality space corresponding to the position andorientation of the head mounted display device, and distributes it tothe head mounted display device. Further, the image processing apparatus104 generates an image of the mixed reality space corresponding to theposition and orientation of the portable device, and distributes it tothe portable device. A method of generating an image of the mixedreality space is the same as those in the above-described embodiments.

For example, a smartphone has an application that superimposes anddisplays AR (Augmented Reality) information on video based on a featureamount (for example, natural feature or QR Code®) detected from an imagecaptured by the image capturing unit. In some cases, orientationinformation of the smartphone itself detected by the orientation sensoris reflected in the AR display form. In such a case, the smartphoneserves as a synchronous control apparatus to synchronize another devicein accordance with the exposure time of the image capturing unit, as inthe above-described embodiments. This can implement high-precisionsynchronous processing.

The HMD 101 and the image processing apparatus 104 may be integrated.Instead of the head mounted display device, the portable device and theimage processing apparatus 104 may be integrated.

In the above-described embodiments, the HMD 101 includes the orientationsensor 503. However, the present invention is not limited to this, andnecessary information may be obtained from an image captured by anobjective camera installed near the wearer of the HMD 101.

Numerical values, arithmetic methods, processing execution timings, andthe like used in the above-described embodiments are merely examples forconcrete descriptions, and it is not intended to limit the embodimentsto these examples.

Some or all of the above-described embodiments may be combined and used.Some or all of the above-described embodiments may be selectively used.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-009447 filed Jan. 23, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A synchronous control apparatus comprising: afirst setting unit configured to set a first offset, wherein the firstoffset is based on a time till a reference of a synchronous timing aftera detection timing of a start signal representing a start timing ofimage output from a first image capturing unit, a time required for asecond image capturing unit to start exposure after receiving a firstsynchronous signal, and an exposure time of one frame in the secondimage capturing unit; a second setting unit configured to set a secondoffset, wherein the second offset is based on the time till thereference of the synchronous timing after the detection timing of thestart signal, and a time required for a sensor to start measurementafter receiving a second synchronous signal; a first supply unitconfigured to, in a case where the start signal is detected, generatethe first synchronous signal after the first offset from the detectiontiming of the start signal to supply the generated first synchronoussignal to the second image capturing unit; and a second supply unitconfigured to, in a case where the start signal is detected, generatethe second synchronous signal after the second offset from the detectiontiming of the start signal to supply the generated second synchronoussignal to the sensor.
 2. The synchronous control apparatus according toclaim 1, wherein the first image capturing unit captures an image by afirst shutter method, and the second image capturing unit captures animage by a second shutter method different from the first shuttermethod.
 3. The synchronous control apparatus according to claim 1,wherein the first image capturing unit is a device having no inputfunction of a synchronous signal.
 4. The synchronous control apparatusaccording to claim 1, wherein when one of an exposure time of the firstimage capturing unit and an exposure time of the second image capturingunit is changed, the first setting unit obtains the first offset basedon the changed exposure time.
 5. The synchronous control apparatusaccording to claim 1, wherein in a case where the time till thereference of the synchronous timing after the detection timing of thestart signal is t61, the time required for the second image capturingunit to start the exposure after receiving the first synchronous signalis t62, and the exposure time of the one frame in the second imagecapturing unit is t63, the first setting unit sets (t61−t62−t63/2) asthe first offset.
 6. The synchronous control apparatus according toclaim 1, wherein when one of an exposure time of the first imagecapturing unit and an exposure time of the second image capturing unitis changed, the second setting unit obtains the second offset based onthe changed exposure time.
 7. The synchronous control apparatusaccording to claim 1, further comprising: an obtaining unit configuredto obtain a composite image of an image of a virtual space generatedbased on an image captured by the second image capturing unit and aposition and orientation measured by the sensor, and an image capturedby the first image capturing unit; and a display unit configured todisplay the composite image.
 8. The synchronous control apparatusaccording to claim 1, wherein the reference of the synchronous timing isone of an exposure start time of a start line of an image captured bythe first image capturing unit, an exposure center time of a center lineof an image captured by the first image capturing unit, and an exposureend time of a final line of an image captured by the first imagecapturing unit.
 9. The synchronous control apparatus according to claim1, wherein the synchronous control apparatus is a head mounted displaydevice which includes the first image capturing unit, the second imagecapturing unit, and the sensor.
 10. The synchronous control apparatusaccording to claim 2, wherein the first shutter method is a rollingshutter method, and the second shutter method is a global shuttermethod.
 11. A method comprising: setting a first offset, wherein thefirst offset is based on a time till a reference of a synchronous timingafter a detection timing of a start signal representing a start timingof image output from a first image capturing unit, a time required for asecond image capturing unit to start exposure after receiving a firstsynchronous signal, and an exposure time of one frame in the secondimage capturing unit; setting a second offset, wherein the second offsetis based on the time till the reference of the synchronous timing afterthe detection timing of the start signal, and a time required for asensor to start measurement after receiving a second synchronous signal;in a case where the start signal is detected, generating, the firstsynchronous signal after the first offset from the detection timing ofthe start signal to supply the generated first synchronous signal to thesecond image capturing unit; and in a case where the start signal isdetected, generating the second synchronous signal after the secondoffset from the detection timing of the start signal to supply thegenerated second synchronous signal to the sensor.
 12. A systemcomprising a synchronous control apparatus and an image processingapparatus, the synchronous control apparatus including: a first settingunit configured to set a first offset, wherein the first offset is basedon a time till a reference of a synchronous timing after a detectiontiming of a start signal representing a start timing of image outputfrom a first image capturing unit, a time required for a second imagecapturing unit to start exposure after receiving a first synchronoussignal, and an exposure time of one frame in the second image capturingunit; a second setting unit configured to set a second offset, whereinthe second offset is based on the time till the reference of thesynchronous timing after the detection timing of the start signal, and atime required for a sensor to start measurement after receiving a secondsynchronous signal; a first supply unit configured to, in a case wherethe start signal is detected, generate the first synchronous signalafter the first offset from the detection timing of the start signal tosupply the generated first synchronous signal to the second imagecapturing unit; and a second supply unit configured to, in a case wherethe start signal is detected, generate the second synchronous signalafter the second offset from the detection timing of the start signal tosupply the generated second synchronous signal to the sensor, and theimage processing apparatus including: an obtaining unit configured toobtain, from the synchronous control apparatus, an image captured by thefirst image capturing unit, an image captured by the second imagecapturing unit in response to reception of the first synchronous signal,and the measurement measured by the sensor in response to reception ofthe second synchronous signal; a unit configured to generate an image ofa virtual space based on the image captured by the second imagecapturing unit and the measurement measured by the sensor, and generatea composite image of the generated image of the virtual space and theimage captured by the first image capturing unit; and a unit configuredto output the composite image to the synchronous control apparatus. 13.The system according to claim 12, wherein the synchronous controlapparatus is a head mounted display device which includes the firstimage capturing unit, the second image capturing unit, the sensor, and adisplay unit configured to display the composite image.
 14. Thesynchronous control apparatus according to claim 1, wherein in a casewhere the time till the reference of the synchronous timing after thedetection timing of the start signal is t61, and the time required forthe sensor to start the measurement after receiving the secondsynchronous signal is t64, the second setting unit sets (t61−t64) as thesecond offset.